Skip to content

feat(budget): add opt-in durable ambiguous usage holds - #44

Merged
ALikesToCode merged 2 commits into
feat/shared-generation-cachefrom
feat/usage-reservations
Oct 9, 2026
Merged

ALikesToCode merged 2 commits into
feat/shared-generation-cachefrom
feat/usage-reservations

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in durable usage holds for dollar-budgeted keys behind USAGE_RESERVATIONS_ENABLED, which defaults to false. With the flag off, accounting, responses, headers and storage are unchanged. Requests without a monetary budget are unchanged even with the flag on.

  • Lifecycle: a reservation moves through reserved → dispatched, then to settled (measured provider cost) or to unknown (completion or usage uncertain).
    • A disconnect after handoff keeps the estimate held.
    • A confirmed cancellation before dispatch, or a cache hit before dispatch, settles at zero.
    • A success with missing billable usage keeps its hold. Missing counters stay null rather than becoming a measured zero.
  • Admission:
    • Every unresolved hold counts against both current UTC budget periods.
    • An estimate must fit every configured budget, otherwise the request gets 429 budget_exceeded.
    • An unpriced eligible model refuses admission with 503 unpriced_reservation.
    • Money is stored as integer units of USD 1e-10. Holds round up and caps round down.
  • Review: USAGE_HOLD_REVIEW_AFTER_SECONDS (default three days) marks an unresolved hold needs_review. It never releases money.
  • Reconciliation: unknown holds need an explicit administrator reconciliation with the current revision, the measured cost, a provider evidence reference and an audit reason. It uses compare-and-swap, keeps the transition history, and rejects conflicting replays.
  • /v1/usage:
    • Reports content-free budget.reservations counters.
    • in_flight_usd includes unresolved holds.
    • If storage is unavailable, the totals report as unavailable and remaining budgets are null; spendable money is never reported in that case.
  • Storage:
    • SQLite uses USAGE_DB_PATH.
    • D1 uses the new tables usage_reservation_budgets, usage_reservations and usage_reservation_transitions (migration 0020) through the private http://intelligence.internal/v1/reservations endpoint.
    • A missing binding, missing tables or a transport failure returns 503 usage_reservations_unavailable before provider submission.
    • Records contain no prompts, responses, credentials or provider bodies.

Details, including the operator steps before first enabling it, are in docs/usage-reservations.md.

Glue left for the Wave 4 integration PR

  • Flask request owners call request_accounting.mark_dispatched immediately before provider submission. The existing cancellation observer also records the handoff.
  • Native owners use createReservationLifecycle.
  • The integration PR injects these call sites, mounts the private /v1/reservations endpoint, and adds the README line. There is no public reconciliation endpoint yet.

Stacking

This PR is based on feat/shared-generation-cache (#43), so migrations 0018–0020 stay contiguous. Review only the top commit, feat(budget): add opt-in durable ambiguous usage holds; the merge commit brings in #43.

Deployment

Apply the migration before enabling the flag. A push to main does not apply D1 migrations:

npx wrangler d1 migrations apply multillm-intelligence --remote

Shared registrations

  • Added tests/test_usage_reservations_worker.mjs to test:worker.
  • Added USAGE_RESERVATIONS_ENABLED and USAGE_HOLD_REVIEW_AFTER_SECONDS to the Container env keys.
  • Added the three reservation tables to the D1 readiness tables.
  • Added worker/reservations-d1.mjs to the secret firewall inventory.

Tests

  • New tests: tests/test_usage_reservations.py and tests/test_usage_reservations_worker.mjs.
  • Full Python suite: 3,621 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,399 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 20 contiguous, additive migrations.
  • test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 4.

Track each budgeted request through reserve, dispatched, settled or unknown, and reconciled states behind USAGE_RESERVATIONS_ENABLED (default off). An unknown completion keeps a conservative estimated hold instead of a zero charge or a timed refund; USAGE_HOLD_REVIEW_AFTER_SECONDS marks old holds for review, and admins reconcile with measured evidence or an audited adjustment. Reservations are stored in D1 (migration 0020).
# Conflicts:
#	package.json
#	worker/container-env.mjs
#	worker/d1-schema.mjs
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 4194733 Oct 09 2026, 12:38 AM

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1da7a0ec-5d60-4955-b22d-29883648b410

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T00:40:18.178605Z 4194733 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 41947335e9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +302 to +304
if current["state"] == "reserved":
cls.mark_dispatched(reservation)
current = store.get(reservation)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Release reservations that never reached dispatch

When a route rejects locally after request_accounting.begin()—for example, payload validation returns 400 before any provider call—the reservation is still reserved, but this branch marks it dispatched and then transitions it to unknown. Because unknown holds never expire, ordinary client errors permanently consume the key's budget and require administrator reconciliation; a still-reserved completion should instead settle at zero because the dispatch hook was never reached.

Useful? React with 👍 / 👎.

Comment on lines +317 to +323
if reservation_store.enabled():
if before_dispatch:
store = reservation_store.get_store()
row = store.get(reservation)
store.transition(reservation, row["revision"], "settled", cost_usd=0, basis="released",
settlement_id=reservation, transition_id=uuid.uuid4().hex)
return

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve pre-dispatch releases for outer accounting

With reservations enabled, settle(reservation) now silently returns unless before_dispatch=True, but services/accounted_dispatch.py:20 still uses the old call when cascades, image quality, and quality-checked media replace an aggregate request with subrequests. Those flows therefore retain the unused outer hold while creating additional holds for each subrequest, eventually exhausting the user's budget even after successful settlement.

Useful? React with 👍 / 👎.

Comment on lines +207 to +208
if reservation_store.enabled():
return cls._reserve_durable(user, estimate_usd, now)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject unpriced gateway-owned subrequests

The durable path only rejects an unpriced request when it receives None, but services/accounted_dispatch.py:43 continues to call estimate_cost(), which converts a missing model price to 0.0. With reservations enabled, an unpriced cascade, image-quality, or shadow-evaluation subrequest therefore receives a zero-dollar reservation and bypasses budget enforcement instead of returning unpriced_reservation as other request paths do.

Useful? React with 👍 / 👎.

store.transition(identity,0,'dispatched',transition_id='${id(2)}',now=now)
store.transition(identity,1,'unknown',transition_id='${id(3)}',input_tokens=0,now=now)
print(json.dumps([store.get(identity),store.summary('alice',now)]))`;
const python = spawnSync("/home/mysterious/storage/github/MultiLLM-Proxy/.venv/bin/python", ["-I", "-c", code], {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Invoke Python through a portable interpreter path

This newly registered worker test hard-codes an author-specific virtualenv path. On any CI runner or checkout not located under /home/mysterious/storage/github/MultiLLM-Proxy, spawnSync cannot start the process, returns a null status, and makes npm run test:worker fail before comparing the contracts; use the repository/environment Python executable instead.

Useful? React with 👍 / 👎.

Comment on lines +15 to +17
export class ReservationError extends Error {
constructor(code = "usage_reservations_unavailable", status = 503) { super(code); this.code = code; this.status = status; }
response() { return reply({ error: { code: this.code, message: "The usage reservation operation could not be completed." } }, this.status); }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Convert reservation failures into native gateway responses

When this lifecycle is registered with nativeGenerationFetch, admission or handoff failures throw ReservationError, but the gateway catch path only converts AdmissionError instances into responses and rethrows every other error. A missing reservation table, unavailable transport, or budget denial will therefore escape as an unhandled worker error instead of the promised bounded JSON 503 or 429 response, even though ReservationError.response() exists.

Useful? React with 👍 / 👎.

Comment on lines +357 to +358
if reservation_store.enabled():
durable = cls._durable_totals(user, now)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip durable status reads for keys without budgets

When the global flag is enabled, BudgetService.status() selects the reservation authority for every principal, even if both monetary limits are unset. Consequently an unbudgeted key's /v1/usage response gains reservation fields and becomes dependent on the new storage—reporting unavailable totals when that storage is down—despite the stated guarantee that keys without monetary budgets remain unchanged.

Useful? React with 👍 / 👎.

Comment on lines +266 to +268
const result = await call({ version: 1, operation: "reserve", id: crypto.randomUUID().replaceAll("-", ""), ...values });
if (result.error) throw new ReservationError(result.error.code, result.error.code === "budget_exceeded" ? 429 : 503);
reservation = result.reservation;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject malformed reservation replies before dispatch

If the injected call transport resolves to a non-error object with a missing or malformed reservation field—for example, an adapter accidentally returns an undecoded Response or a truncated contract object—admission assigns undefined and returns successfully. before_dispatch() then becomes a no-op, so the provider request proceeds without any durable hold; validate the version and reservation shape and fail closed as the Python D1 client does.

Useful? React with 👍 / 👎.

@ALikesToCode
ALikesToCode merged commit 59e6e45 into feat/shared-generation-cache Oct 9, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant